Recurrent Neural Network Training with the Extended Kalman Filter

Recurrent neural networks, in contrast to the classical feedforward neural networks, better handle inputs that have spacetime structure, e.g. symbolic time series. Since the classic gradient methods for recurrent neural network training on longer input sequences converge very poorly and slowly, the alternative approaches are needed. We describe the principles of the training method with the Extended Kalman Filter and with its modifications Unscented Kalman Filter, nprKF and with their joint versions. The Joint Unscented Kalman Filter was not used for this purpose before. We compare the performance of these filters and of the classic Truncated Backpropagation Through Time in an experiment for nextsymbol prediction – word sequence generated by Reber automaton. All the new filters achieved significantly better results.